The present invention relates to methods of correcting distortions generated in a taken image as distortions, which are caused by a hand movement when the image is taken by using an image-taking apparatus, and relates to apparatus for correcting such distortions by adoption of the methods. The present invention can be well applied to an image-taking apparatus such as a video camera or a digital still camera and a recording/reproduction apparatus having an image-taking function.
In an electronic image-taking apparatus capable of storing a image taken electronically in an image-taking device and reading out the image from the device, when the position of the image-taking device physically changes at a relatively high velocity in the horizontal and/or vertical direction of the image being taken in the course of the operation to take the image due to, for example, the so-called movement of the hand of the photographer tremble, the fact that the image is being taken at an unstable place like a place on a ship or the like, a distortion is generated in the image being taken.
In the case of an electronic image-taking apparatus, however, data of a taken image read out from the image-taking device is subjected to digital processing to correct distortions so that taken-image distortions attributed to a hand movement or another cause can be corrected.
In the conventional image-taking apparatus such as a video camera or a digital still camera and the conventional recording/reproduction apparatus having an image-taking function (or the conventional cell-phone having a camera or the conventional personal computer having a camera), as the image-taking device, a solid-state image-taking device employing a CCD (Charge Couple Device) is widely used in general. In the following description, the solid-state image-taking device employing a CCD is referred to as a CCD imager.
As described in documents such as Japanese Patent No. 3384459, which is used as patent document 1 in this specification, a large number of technologies for correcting distortions caused by a hand movement on the assumption that a CCD imager is used as an image-taking device has been proposed. As a matter of fact, products each adopting a technology for correcting distortions caused by a hand movement are widely sold in world markets. The technology for correcting distortions caused by a hand movement is referred to hereafter simply as a hand-movement correction technology.
The conventional hand-movement correction technology makes use of a characteristic that a time to sample the quantity of light accumulated in all pixels of the CCD imager is uniform throughout the structure of the CCD imager. That is to say, a sampling operation is carried out once per frame.
In the CCD imager, all pixels are exposed at the same time and the whole image data of 1 frame is fetched with just a single timing. Thus, for all pixels of 1 frame, only one hand-movement displacement Vcs shown by an arrow in FIG. 18 can be considered. That is to say, a taken-image distortion caused by a hand movement as a distortion of a frame of a taken image can be corrected by correction of a reading pixel position (or a sampling pixel position) by the hand-movement displacement Vcs, which can be detected for the frame as a distortion caused by a displacement from an area Fla enclosed by a solid line in FIG. 18 as the accumulation area of the original frame of the taken image to an area FLb enclosed by a dash line in the image due to the hand movement.
It is to be noted that, in general, not all pixels of an image-taking device are treated as effective pixels as shown in FIG. 18. Instead, only pixels in a central part of the entire area AFL are used as a valid image area EFL. The entire area AFL is referred to as an effective image area including all pixels. Defined by a vertical valid length and a horizontal valid length, the valid image area EFL is obtained by excluding peripheral areas from the effective image area AFL.
With such an imager used, by using only data of pixels originally owned by the imager, it is possible to correct a distortion, which is resulted in when the reading pixel position changes due to a hand movement, provided that the magnitude of the hand movement is within a range smaller than the difference between the effective image area AFL and the valid image area EFL. Thus, in comparison with interpolation processing or the like in which data necessary for correction of a distortion caused by a hand movement is generated, the amount of image deterioration decreases.
In recent years, as the image-taking device of an electronic image-taking apparatus, a solid-state image-taking device having an X-Y address type is employed. The solid-state image-taking device having an X-Y address type is an image-taking device capable of outputting data of any pixel unit on the screen by specifying the horizontal-direction and vertical-direction positions of the pixel. The horizontal-direction position is also referred to as an X-direction position and the vertical-direction position is also referred to as a Y-direction position. An example of the solid-state image-taking device having an X-Y address type is a CMOS-type solid-state image-taking device, which is referred to hereafter as a CMOS imager.
The CMOS imager has the following features:
(a) The CMOS imager is an imager of an amplifying type. Since an amplified signal is read out, the imager offers a high sensitivity
(b) Since a CMOS circuit is employed, the power consumption is low
(c) The CMOS imager can be manufactured at a low cost
(d) As a rule, the CMOS imager is capable of outputting data of any pixel selected at random or making an access to any pixel selected at random.
Thus, even though the CMOS imager is capable of outputting data of a taken image in pixel units, in practicality, the CMOS imager generally outputs (or samples) the data of a taken image in pixel-group units each including pixels on one horizontal line in a process to output the data.
If the CMOS imager outputs data of a taken image in horizontal-line units as described above, as shown in FIG. 19, there is a time shift Δt between exposure times of adjacent horizontal lines. The time shift Δt is equal to a difference between in reading time between the adjacent horizontal lines. It is to be noted that, if the CMOS imager outputs data of a taken image in pixel units, a difference in reading time between adjacent pixels is so smaller than the difference between in reading time between adjacent horizontal lines so that the difference between in reading time between adjacent pixels can be ignored. Even if the CMOS imager outputs data of a taken image in pixel units, however, there is still a similar time shift between exposure times.
Because of what is described above, in the case of an image-taking apparatus employing a CMOS imager, for an image that should naturally be obtained as shown in FIG. 20A as an image showing a vertically erected house and a vertically erected tree, when the image is taken from a position inside a running train, for example, an image of a slanting house and an inclined tree is actually obtained as shown in FIG. 20B. The difference between the images shown in FIGS. 20A and 20B is a result of the so-called focal plane phenomenon peculiar to the CMOS image.
The typical image shown in FIG. 20B is an image, which is taken while the image-taking apparatus is moving in the horizontal direction. If the image of a photographing object is taken while the image-taking apparatus is moving in the vertical direction, however, the resulting image will show the object shrunk or expanded in the vertical direction as shown in none of the figures.
The phenomenon described above occurs strikingly when the relative velocity between the object of photographing and the image-taking apparatus employing the CMOS imager. For example, the phenomenon described above occurs strikingly when the photographer takes an image of the photographing object by using the image-taking apparatus while moving at a high velocity or, conversely, when the photographer staying at a fixed location takes an image of the photographing object, which is moving at a high velocity. In the case of ordinary photographing, however, such a situation can be said to be a situation that occurs rarely.
When the photographer takes an image of the photographing object by using the image-taking apparatus held in its hand, however, the hand may move a little bit at a high velocity. In this case, the focal plane phenomenon described above occurs due to the hand movement.
This is because the value of a hand movement happening to a CMOS imager is not one value in a frame as is the case with a CCD imager but, as described above, a value varying from pixel to pixel or horizontal line to horizontal line due to differences in sampling time between the pixels in the frame or between the horizontal lines in the frame. For this reason, in an image-taking apparatus employing a CMOS imager, a distortion caused by the focal plane phenomenon described above remains uncorrected even if a correction process merely using a hand-movement quantity for one frame is carried out. In the following description, a hand movement happening to a CMOS imager is also referred to as a CMOS hand movement.
In the case of an image-taking apparatus employing a CMOS imager, a queer squashy distortion like one shown in FIG. 20C is observed in the output of a taken image of a photographing object in the event of a hand movement due to the fact that the direction, magnitude and velocity of the hand movement are not uniform in a frame or the taken image.
In the case of an apparatus for taking a still image, as preconditions for still-image photographing, it is assumed that the magnitude of a hand movement is limited and a mechanical shutter is employed so that a focal plane phenomenon caused by a CMOS hand movement can be avoided relatively with ease. An example of the apparatus for taking a still image is a digital still camera.
In the case of another kind of image-taking apparatus such as a video camera, on the other hand, a moving-image photographing is assumed. Thus, a moving-picture-taking apparatus for business applications or for a moving-picture-taking apparatus of a high-class category adopts a method for virtually preventing a focal plane phenomenon from occurring due to a CMOS hand movement. In accordance with this method, a maximum difference in sampling time in one frame is reduced and the data of a taken image is read out at a very high velocity.
In addition, the magnitude of a hand movement relative to an object of photographing rises as the magnification of an optical zoom is increased. Thus, if the image-taking apparatus does not have an optical-zoom function or the image-taking apparatus has an optical-zoom function but the magnification of the optical zoom is small, the CMOS hand movement does not raise a big problem even in an application of taking moving images. In the first place, for an inexpensive image-taking apparatus where even a distortion caused by a hand movement is not corrected by using an acceleration sensor as is the case with the conventional image-taking apparatus employing the CCD imager, a bad effect caused by the CMOS hand movement is relatively small, not even raising a problem in many cases.
In order to solve the problems described above, there has been demanded a technology for correcting and avoiding a distortion of a focal plane phenomenon caused by a CMOS hand movement as a technology that can be implemented with a high degree of precision at a low cost in an image-taking apparatus having a high-magnification optical zoom function for taking moving images as a main application by carrying out digital-signal processing and with neither employing a special device nor utilizing a high-velocity sampling clock signal.
Japanese Patent Laid-open No. 2004-266322 serving as patent document 2 describes a method to correct a distortion including a focal plain phenomenon caused by a CMOS hand movement in a taken image.
In accordance with the distortion correction method described in patent document 2, when data of a taken image is read out from the CMOS imager in horizontal-line units, a positional displacement quantity caused by a generated hand movement for each horizontal line is detected and a correction is made so as to read out the data of the taken image from a position shifted from the original position by the detected positional displacement quantity in a direction opposite to the direction of the hand movement. The positional displacement quantity is also referred to hereafter as a hand-movement quantity.
However, this method raises a problem of a difficulty to obtain a hand-movement quantity for each horizontal line as a problem caused by conditions such as the sampling frequency of a sensor for detecting a hand movement. In order to solve this problem, in accordance with the distortion correction method described in patent document 2, a hand-movement quantity is detected discretely in the vertical direction of the screen every plurality of horizontal lines as shown in FIG. 21A. In the example shown in FIGS. 21A, 21B, and 21C, hand-movement quantities Q1, Q2, Q3 and so on are detected every 50 horizontal lines as shown in FIG. 21B. It is to be noted that the figure shows only hand-movement quantities in the horizontal direction.
The hand-movement quantities of the remaining 49 horizontal lines not directly subjected to the process to detect hand-movement quantities, that is, the hand-movement quantities of the horizontal lines other than the horizontal line subjected to the process to detect hand-movement quantities, are found by interpolation based on the hand-movement quantities Q1, Q2, Q3 and so on. As the interpolation method, it is possible to adopt some techniques, one of which is shown in FIG. 21C. In accordance with the interpolation method shown in FIG. 21C, basically, the interpolation to find a hand-movement quantity of any specific one of 49 horizontal lines not subjected to the process to detect hand-movement quantities is based on two hand-movement quantities Qn and Qn+1 where notation n in the subscript denotes an integer at least equal to 1. The hand-movement quantity Qn is the hand-movement quantity of a horizontal line immediately preceding the 49 horizontal lines while the hand-movement quantity Qn+1 is the hand-movement quantity of the a horizontal line immediately succeeding the 49 horizontal lines.
In accordance with interpolation method (1) shown in FIG. 21C, for example, the detected hand-movement quantity Qn of the horizontal line immediately preceding the 49 horizontal lines is used as it is for the horizontal in the first half of the 49 horizontal lines. On the other hand, the detected hand-movement quantity Qn+1 of the horizontal line immediately succeeding the 49 horizontal lines is used as it is for the horizontal in the second half of the 49 horizontal lines.
In accordance with interpolation method (2) shown in FIG. 21C, on the other hand, values represented by a straight line connecting the detected hand-movement quantity Qn of the horizontal line immediately preceding the 49 horizontal lines and the detected hand-movement quantity Qn+1 of the horizontal line immediately succeeding the 49 horizontal lines are used as hand-movement quantities of the 49 horizontal lines. That is to say, interpolation method (2) is the so-called average value interpolation method.
By adoption of any of the distortion correction methods described in patent document 2, it is possible to correct a distortion including a focal plain phenomenon caused by a CMOS hand movement in an image taken by using a CMOS imager.
In accordance with the distortion correction methods described in patent document 2, however, hand movements are detected at discrete locations to find hand-movement quantities at the discrete locations on the basis of results of the detection and the hand-movement quantities at the discrete locations are used in an interpolation process to infer a hand-movement quantity for any horizontal line between the discrete locations.
In consequence, in the case of the distortion correction methods described in patent document 2, the hand-movement quantity for a horizontal line between the discrete locations cannot be found in the interpolation process before both the hand-movement quantity Qn at the discrete location preceding the horizontal line the hand-movement quantity Qn+1 at the discrete location succeeding the horizontal line are found. Thus, the process to find the hand-movement quantity for a horizontal line following the preceding discrete location is carried out at a time delayed from the preceding discrete location by up to a time delay corresponding to a sampling period, which is determined by the number of horizontal lines between the preceding discrete location and the succeeding discrete location. This is because the sampling period is a sum of discrete horizontal-line periods each representing a period between two consecutive horizontal lines.
If interpolation method (1) described above is adopted, it is quite within the bounds of possibility that the picture position is shifted due to the varying time of the hand-movement quantity found as an interpolated value.
In accordance with interpolation method (2), the rate of change in hand-movement quantity is found from the hand-movement quantity Qn detected at the immediately preceding position and the hand-movement quantity Qn+1 detected at the immediately succeeding position. Then, the hand-movement quantity of a specific horizontal line is found by multiplying the rate of change in hand-movement quantity by the number of horizontal lines existing between the immediately preceding position and the specific horizontal line. Thus, a multiplier is required and it is necessary to provide a register used for separately setting a parameter of multiplication. As a result, the hardware becomes complicated and the size of the circuit increases. In addition, because of a multiplication error, the sampling value of the hand-movement quantity at a border position, at which the hand-movement quantity changes, does not vary continuously as is the case with interpolation method (2) shown in FIG. 21C. It is therefore quite within the bounds of possibility that the image position is shifted.